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Foreword 


The adoption of the Bologna process has influenced the development of quality 
assurance across many countries in Europe. In particular, the implementation of the 
Framework for Qualifications in the European Higher Education Area has stimulated 
discussion about the three cycle model, which uses generic descriptors for each cycle 
based on learning outcomes and competences. This in turn has led both agencies and 
institutions to seek to further define and understand the terms ‘learning outcomes’ and 
‘competencies’ for use in their own national situations. 

This trailblazing report by the Nordic Quality Assurance Network in Higher 
Education makes a valuable contribution to expanding agencies’ knowledge and 
understanding of the concept of learning outcomes, by examining the Nordic 
perspective. It outlines how the Nordic higher education institutions are currently 
working on developing and implementing learning outcomes, and focuses on 
preparations under way at the quality assurance agencies for future work to evaluate 
those outcomes. 

The value of learning outcomes has been recognised by all players - agencies, 
institutions, students, ministries - but in the future, measuring learning outcomes may 
prove to be both the greatest challenge, and the greatest reward. 

I hope this report will inform and stimulate debate on this important topic amongst 
ENQA members and other stakeholders. 

Peter Williams 

President 

ENQA 


Preface 


The demands of the Bologna-process are influencing higher education in the Nordic 
countries. All five Nordic countries Denmark, Finland, Iceland, Norway and Sweden 
have adopted the qualification framework in the European higher education area, 
comprising three cycles with generic descriptors for each cycle based on learning 
outcomes and competences, which will be implemented by 2010. The higher education 
institutions are now working on implementing, describing, measuring and evaluating 
learning outcomes. The quality assurance agencies are preparing for future evaluations 
including a learning outcomes component. 

The Nordic Quality Assurance Network in Higher Education (NOQA) is a forum 
for disseminating information, sharing experience and pursuing projects of mutual 
interest. The main objective is to create a joint understanding of different Nordic 
viewpoints on issues related to higher education quality assurance. 

The Nordic network has engaged in seven previous joint projects, which have all 
encouraged and produced new perspectives, a broader understanding of different 
viewpoints and the exchange of experience and information related to the work 
with quality assurance. This time NOQA wanted to take a look at something not yet 
established, something in a process of development. The working title was: “How 
learning outcomes (National and European Qualification Framework) are or may be an 
important basis for evaluation criteria for study programmes.” The work on national 
qualification frameworks and learning outcomes is at different stages in the various 
Nordic countries, which makes this starting point a challenge particularly when 
most of the Nordic quality assurance agencies are at the present time in a process of 
discussing how learning outcomes are a part of the agencies’ evaluations. The Nordic 
higher education institutions are working on developing and implementing learning 
outcomes. This report focuses on the Nordic quality assurance agencies’ preparations 
for the future work of evaluating learning outcomes, and also includes examples from 
the work undertaken at higher education institutions. 

The aim of this joint Nordic project is to expand the agencies’ knowledge and 
understanding of the concept of “learning outcomes”. The hope of the project group 
is to inspire the Nordic quality assurance agencies, the higher education institutions 
and others that are interested in the process of implementing learning outcomes and 
developing methods of evaluation. 

For further reading on the development of learning outcomes, the report from the 
United Kingdom Bologna seminar can be recommended (Bologna Seminar 1-2 July 
2004) and for the implementation of Bologna, the EUA Bologna Handbook (The EUA 
Bologna Handbook, 2006). 

We would like to express our warmest thanks to all institutions participating in this 
project. 


Summary 

All the Nordic countries have, in connection with the Bologna-process, an obligation 
to develop a national qualification framework by 2010. Learning outcomes are for 
all the countries an important part of the national qualification framework. The 
implementation of learning outcomes and a national qualification framework is at 
different stages in the five countries. The national qualification framework in Iceland 
is already implemented. In Denmark the first qualification framework was approved 
in 2003, and in 2008 a new second generation qualification framework is being 
developed. In Norway and Sweden the proposal of a national qualification framework 
has been to hearing with the institutions and stakeholders, and the final work is now 
in progress. In Finland the Ministry will most likely form a working group to create a 
national qualification framework in 2008-2009. 

In all Nordic countries higher education institutions have started the work with 
developing and formulating learning outcomes for their study programmes, subjects 
and modules. All higher education institutions in Iceland and Sweden, and the Finnish 
polytechnics have introduced learning outcomes. In Denmark all institutions have 
been working with learning outcomes, but not all have followed the terminology of 
the national qualification framework. In Norway, the institutions are formulating 
descriptions of the study programme objectives that are quite similar to descriptions 
of learning outcomes, and some of the institutions have taken the initiative to describe 
more outcome related descriptions of their own study programmes. 

The majority of the participating institutions found learning outcomes to be 
very useful. Among the primary strengths is the aspect that the learning outcomes 
provide better information on courses and programmes. Also, the encouragement to 
develop teaching methods and an improvement of the connection between content, 
examination and assessment was seen as useful. The discussions between teachers and 
students on learning outcomes have also reportedly created better understanding of the 
subject, content and attitudes. Only a few institutions have used learning outcomes to 
probe working-life expectations and future developments. None of the institutions use 
learning outcomes as a part of their quality assurance system yet, but, there is reason 
to expect their more ambitious application as a tool to assess and develop teaching and 
institutional performance in the future. 

However, the Nordic quality assurance agencies and the Icelandic Ministry of 
Education, Science and Culture have limited experiences of evaluating learning 
outcomes. Denmark and Norway have since 2003 carried out evaluations resembling 
learning outcomes. These experiences of evaluating learning outcomes are part of an 
accreditation-process. 

For quality assurance agencies, the focus on learning outcomes is at an aggregated 
level, such as study programmes or institutions, and not at the individual student 
level. There are inherent challenges with the agencies evaluation of learning outcomes 
in quality audits and programmes, subjects or modules. In evaluations, these are, 
for example, connected with the assessment criteria, ways of measuring learning 
outcomes, and different interpretations of the contexts in the different countries. One 
challenge is to develop indicators that measure the achievement of learning outcomes 


compared with the intended learning outcomes, and not easily measurable like 
efficiency, etc., that provides limited information about the learning outcomes. 

It seems that learning outcomes can be integrated relatively easy into quality audits 
if higher education institutions have implemented learning outcomes, in their study 
programmes and they are a part of the institution’s quality assurance system. The 
institutions themselves don’t really know yet how they are going to evaluate whether 
the students are fulfilling the learning outcomes and how they make sure that the 
learning outcomes are in accordance with the descriptions of the national qualification 
framework within the programmes. 

Currently, higher education institutions, quality assurance agencies and research 
institutes worldwide are working on the topics of describing and formulating, 
measuring and evaluating learning outcomes. In the future, we will find a range of 
examples of assessment criteria and measurement indicators and different approaches 
on evaluating learning outcomes. Both audit and evaluation of study can promote 
enhancement of the work on learning outcomes. 


PART A. Introduction 


Project description 

The objective of this project is to expand the agencies’ knowledge and understanding of 
the concept of “learning outcomes” in relation to their work with the quality assurance 
of higher education. 

The project aim is to describe and analyse the implementation and use of learning 
outcomes in both the Nordic higher education institutions and the Nordic quality 
assurance agencies, and consider some possible consequences to the agencies’ external 
quality assurance. 

The project attempts to answer the following questions: 

• Why learning outcomes? 

• What is the present situation in each Nordic country regarding 

- national and institutional understanding of learning outcomes 

- the development and use of learning outcomes 

- the aim of using the learning outcomes 

- the impact of using learning outcomes 

• How can we as quality assurance agencies use learning outcomes in evaluations in 
the future? 

- evaluating learning outcomes in higher education audit 

- evaluation learning outcomes in study programmes 

- method/criteria used at other agencies 

Organisation of the work - method 

A project group with members from each of the Nordic quality assurance agencies has 
carried out the project. Gunn Gallavara was the coordinator of the project. The project 
group members have been: 

• Gunn Gallavara, The Norwegian Agency of Quality Assurance in Education 
(NOKUT) 

• Einar Hreinsson, Ministry of Education, Science and Culture, Iceland 

• Matti Kajaste, The Finnish Higher Education Council (FINHEEC) 

• Eric Lindesjoo, The Swedish National Agency for Higher Education (HSV) 

• Christel Solvhjelm, The Danish Evaluation Institute (EVA) 

• Anne Karine Sorskar, The Norwegian Agency of Quality Assurance in Education 
(NOKUT) 

• Mehdi Sedigh Zadeh, The Swedish National Agency for Higher Education (HSV) 

The group had four meetings, the first in Oslo in October 2007. There was a meeting 
in Copenhagen in January and in Helsinki in March 2008, and the last meeting was 
in Reykjavik in April. All members of the group have participated in the writing of the 
report, but the final version has been edited by NOKUT. 

The project group has gathered information from selected higher education 
institutions in each of the Nordic countries regarding their work and experience 
with learning outcomes. The institutions were asked to answer a set of questions in 
order to obtain comparable material. All agencies used a questionnaire to collect the 
information. In addition, EVA, also carried out telephone interviews. 


The institutions have not been selected randomly, and therefore they don’t provide a 
representative picture of the situation concerning learning outcomes in the different 
countries’ higher education institutions. The institutions provide examples of trends 
and work with learning outcomes at higher education institutions. The selected 
institutions are: 

DENMARK: 

University of Aarhus 

Roskilde University 

The Danish School of Design 

Roskilde Business College 

The School of Nursing in Silkeborg 

FINLAND: 

Laurea polytechnic 
Jyvaskyla polytechnic 
Sibelius Academy 

ICELAND 

University of Iceland 
Reykjavik University 
University of Akureyri 

NORWAY 
University of Oslo 
University of Tromso 
Norwegian School of Theology 

SWEDEN 

University of Gothenburg 
Karolinska Institute 
Malmo University 

Furthermore the group also collected information from each of the Nordic quality 
assurance agencies and the Icelandic Ministry of Education, Science and Culture 
regarding their work and experiences with learning outcomes. The findings from each 
of the countries are described in detail in the country reports, but also summed up later 
in part B to show the trends in the Nordic countries. 

Definitions and concepts 

The report is based on the definition used on the official Bologna website: “Learning 
outcomes are statements of what a learner is expected to know, understand and/or be 
able to do at the end of a period of learning”. 

Evaluation is used as a general term when writing about different evaluation 
methods such as accreditation, audit, benchmarking, and evaluation for quality 
enhancement and so on in order to promote quality assurance. 


A distinction may be drawn between internal and external quality assurance. 
Internal quality assurance is the institutions own measures to promote quality in 
their programmes. External quality assurance refers to processes like evaluation, 
accreditation and audits which the agencies undertake in order to investigate the 
quality of a higher education institution, and which is described in the ENQA 
document “Standards and Guidelines for Quality Assurance of the European Higher 
Education Area” published in 2005. 


Why learning outcomes? 

One of the purposes of the Bologna process was to encourage European cooperation 
in quality assurance of higher education with a view to developing comparable criteria 
and methodologies. Qualification frameworks are important instruments in achieving 
comparability and transparency. Qualification frameworks describe the qualifications 
of an education system and how they interlink. National qualification frameworks 
encompass all education qualifications - or all higher education qualifications in an 
education system. They describe what learners may be expected to know, understand 
and be able to do on the basis of a given qualification (learning outcomes) as well as 
how learners can move from one qualification to another within a system. Qualification 
frameworks thus focus on outcomes as much as or more than procedures, and various 
learning paths - including lifelong learning (The Bologna official website). 

Learning outcomes were not mentioned in the original 1999 Bologna Declaration or 
the Prague Communique of 2001, but on the way through the process, the significance 
of a generic way describing the output of education emerged. At the 2005 Bologna 
summit in Bergen, an overarching European Qualifications Framework (EQF) for 
higher education was agreed. This consists of three cycles of generic qualifications (plus 
an optional short cycle, within or linked to the first cycle). It sets out defined generic 
descriptors for each cycle based on learning outcomes and competences (described in 
terms of knowledge, skills and competence and genereic skills) and for the first and 
second cycles, an estimate of the number of credits required. Learning outcomes are 
extensively referred to in various Bologna-related documents including more recently 
the Berlin Communique itself (Berlin Communique 2003): 

Degree structure: ‘Ministers encourage the member states to elaborate a framework of 
comparable and compatible qualifications for their higher education systems, which should 
seek to describe qualifications in terms of workload, level, learning outcomes, competences 
and profile. They also undertake to elaborate an overarching framework of qualifications 
for the European Higher Education Area. 

Within such frameworks, degrees should have different defined outcomes. First and 
second cycle degrees should have different orientations and various profiles in order to 
accommodate a diversity of individual, academic and labour market needs. First cycle 
degrees should give access, in the sense of the Lisbon Recognition Convention, to second 
cycle programmes. Second cycle degrees should give access to doctoral studies. 

Learning outcomes are, in a way, a tool to describe and define a learning and 
assessment process and its product, which can lead to improved pedagogical practice in 
education and improved student learning practice. They place focus on the coherence 
and aims of the qualification, the judgement of the designer and how the qualification 
fits within the traditions of the discipline. By using learning outcomes to describe 
the learning process and its outcomes, institutions are addressing the interests of the 
students and the stakeholders in a learner- centred way rather than being a teacher- 
centred. By that it can be said that the intended learning outcomes are statements 
that predict what learners will have gained as a result of learning. From the students 
perspective, the outcome approach communicates what they are expected to be able to 
do and the criteria that will be used to assess them. This means that the description of 
the learning process has shifted from input to output. 


Learning outcomes outside 
the Nordic countries 


The received information about the exact state of introduction and implementation 
of learning outcomes from all the 46 Bologna countries is very scarce and to some 
extent unreliable. The 2007 review reveals that there is considerable activity across 
Europe, which can be viewed as a positive European-wide movement toward the 
adoption of learning outcomes. However, progress is slightly slow both in the national 
and institutional adoption and the implementation of learning outcomes (UK Bologna 
seminar on learning outcomes, 2004). Despite this, Adam argues in his report (2004) 
that “this is not a negative situation as learning outcomes are part of a massive reform 
package that spans enormous structural and process changes from macro to micro 
levels, encompassing qualification frameworks, quality assurance, institutional and 
curriculum reform. Such innovations require careful and slow implementation.” 

It appears that the Bologna process definition of learning outcomes is commonly 
adopted as it is widely used 1 . However, it is not safe to assume that the detailed practical 
application of learning outcomes is understood in the same way in every country. 

There are possible confusions between, for example, the level of the description of 
learning outcomes, competence and student workload 2 . Detailed experience of learning 
outcomes is limited to just a few countries at both the institutional and national 
levels. The most highly developed systems - that use learning outcomes as a basis 
for the qualification framework, level descriptors, generic qualification descriptors, 
subject descriptors and the individual modules - exist in Scotland (http://www.qaa. 
ac.uk/scotland) and Eire (http://www.hetac.ie/) 3 . According to Vincent, “Learning 
outcomes had fundamentally changed the Scottish sector’s approach to learning since 
the 1990’s and had resulted in enhanced coherence of the learning experience, greater 
transparency, increased dialogue with stakeholders, more opportunity for students to 
manage their own learning and better support for transitions into and out of learning 
programmes at points that suited the needs of the student.” (Bologna seminar on 
learning outcomes based higher education, 2008). 

In addition to Scotland and Eire, England, Wales and Northern Ireland have 
well-established systems that have pioneered the use of learning outcomes in higher 
education (Bologna seminar on learning outcomes based higher education, 2008). 
Belgium, Croatia, Estonia, Hungary, Italy, Moldova, Portugal, Romania, Spain, and 
Switzerland are making rapid progress towards a more comprehensive implementation 
of learning outcomes. Belgium (Flemish Community), Hungary, Ireland, Italy, 

Slovak Republic, Spain, and the UK have developed (or are at the advanced stages of 
implementing) integrated systems that employ learning outcomes approaches at all 
levels of educational activity. There is modest development in the area of learning 


1 “Learning outcomes are statements of what a learner is expected to know, understand and/or be able to demonstrate at the end of 
a period of learning”. S. Adams in Bologna seminar 2008. 

2 Further reading at the final seminar report Bologna seminar 2008 http://www.ond.vlaanderen.be/hogeronderwijs/bologna/ 
BolognaSeminars/documents/Edinburgh/Edinburgh_Feb08_final_report.pdf 

3 Bologna seminar on learning outcomes based higher education (2008): The Scottish experience. Stephen Adam (UK Bologna 
expert). 


outcomes in Estonia, Greece, Lithuania and Latvia (Bologna seminar on learning 
outcomes based higher education, 2008). 

It is clear that activity outside Europe has been taking place notably in Australia, 
New Zealand, South Africa and the USA. 

In the USA Alverno College in Milwaukee, Wisconsin has many years of experience 
of applying learning outcomes. The US engineering accreditation agency known as 
ABET (Leadership and Quality Assurance in Applied Science, Computing, Engineering, 
and Technology Education) has fairly wide resources available online concerning the 
creation of learning outcomes assessment methods. According to ABET, when assessing 
learning outcomes, one asks the following questions: what have students learned, what 
skills have they gained and what attitudes have they developed? It is these questions 
institutions should attempt to answer in collecting relevant data on their performance 
related to learning outcomes. “The higher education institutions must have a 
performance criteria of specific, measurable statements identifying the performance(s) 
required to meet the outcome; confirmable through evidence”. Hence it is important to 
have a desired target level set to which results can be compared (www.abet.org). 

ABET describes a variety of assessment methods for observing learning outcomes: 
written questionnaires, interviews, standardised tests, locally developed tests, 
external evaluations, peer and 360-degree reviews and so on. They all have different 
characteristics related to relevance, accuracy and utility. Some require more effort to 
perform but also provide more accurate information, whereas others are lighter tools. 
No single method is good for measuring a variety of student abilities, but a selection of 
indicators and measurement tools seems necessary. 

The Australian Qualifications Framework provides a comprehensive, nationally 
consistent framework for all post compulsory education and training, which was 
introduced in 1995 (UK Bologna Seminar on learning outcomes, 2004). 


PART B. Learning outcomes in 
the Nordic countries 


This part of the report gives an introduction to the implementation and development 
of learning outcomes in higher education institutions and quality assurance agencies in 
the Nordic countries. 

The information is based on questionnaires to the quality assurance agencies and 
the Icelandic Ministry of Education, Science and Culture. Selected higher education 
institutions also answered a questionnaire on implementing learning outcomes. All 
the participating Nordic higher education institutions were selected because they had 
commenced formulating learning outcomes, but some to a greater extent than others. 

In order to provide a picture of the situation regarding learning outcomes in the 
Nordic countries, each national quality assurance agency contacted a number of higher 
education institutions and asked them to answer a number of questions regarding their 
work and experience with learning outcomes. This resulted in five national pictures 
of higher education institutions and learning outcomes. It is important to stress that 
the national pictures are not representative for all higher education institutions in the 
different countries and simple comparisons between countries are not meaningful. This 
chapter does not present a complete picture of how Nordic higher education institutions 
are working with learning outcomes. Instead, the intention is to try to highlight trends 
within this field. 


years 


The Danish higher education system 

There are three types of institutions offering higher education in Denmark: 

1. Academies of professional higher education or vocational colleges which offer 
2-year academy profession programmes in fields such as business, technology, 
and IT. There are 15 academies of professional higher education and a number of 
vocational colleges. 

2. University colleges which offer 3-4 year professional bachelor programmes in 
fields such as teacher training, business, health, engineering and social work. 
There are 8 university colleges in Denmark. 

3. The universities which offer three-year bachelor programmes, two-year 
candidatus programmes and three-year PhD programmes. There are 8 universities 
in Denmark. 

The different programmes underlined above can be found in the following diagram 
over higher education in Denmark: 

THE DANISH HIGHER EDUCATION SYSTEM 




Bachelor's (BA/BSc) 
programmes 


+ UNIVERSITY SECTOR 





Academy Profession 
programmes 


!_ 

+ COLLEGE SECTOR | 


Master's 
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Diploma 

programmes 

VVU 

programmes 

Adult 

education 


General and vocationally oriented upper secondary education 


Vocational education and training 


Fig 1 Illustration of the Danish higher education system 

In Denmark responsibility for higher education is divided between different ministries 4 . 


4 The Ministry of Science, Technology and Innovation is responsible for higher education in the university sector (bachelor 

candidatus and PhD). The Ministry of Education is responsible for higher education in the college sector (academy profession and 
professional bachelor programmes). In addition to this the Ministry of Culture is responsible for the cultural and creative higher 
educational programmes, the Danish Maritime Authority is responsible for the maritime higher education programmes, the 
Ministry of Defence is responsible for higher education programmes within its field, and the Ministry of Justice is responsible for 
higher education programmes within its field. 











The Danish national qualification framework 

The first Danish qualification framework for higher education was developed in 
2001-02 and approved by the ministers responsible for higher education in 2003 5 . It 
was developed by the Danish Bologna follow-up group and all stakeholders relevant to 
higher education were represented. In short, the national qualification framework of 
2003 focused on the following three elements: 

• a description of the overall/general competency profile for a particular degree. 

• a set of competency goals focusing on the intellectual - , academic - and practical 
competences. 

• a description of the formal aspects (i.e. admittance requirements, length of study 
and further education options) 

From 2003 to 2006, the Danish universities worked towards describing their 
programmes using outcome descriptors. Some of them used the terminology from the 
Danish qualification framework, others developed their own terminology. At the same 
time six pilot projects were carried out at academy profession programmes (AP) and 
adult further education programmes (VVU) and professional bachelor programmes and 
diploma programmes in an attempt to implement the qualification framework and its 
terminology in the Danish college sector. All the experiences of the institutions were 
gathered and used in the revision and development of a second generation qualification 
framework. 

In May 2007 a draft for a new Danish qualification framework was presented. Again, 
a broad group of stakeholders had been involved in the development of the framework. 
The new Danish qualification framework consists of three elements: 

• description of learning outcomes, sub-divided into three main categories: 

- knowledge (with two sub-categories: field of knowledge; level of understanding 
and reflection) 

- skills (with three sub-categories: the character of the skills; assessment and 
decision making; dissemination) 

- competences (with three sub-categories: field of action; co-operation and 
responsibility; learning) 

• description of levels (four levels: sub-degree level; bachelor level; candidatus/ 
master level and PhD level) 

• description of degrees (eight degrees: vocational academy degree; adult further 
education degree; professional bachelor degree; diploma graduate degree; bachelor 
degree; candidatus degree; master degree and PhD degree). 

The new national qualification framework was sent to hearing, and at the moment 
a working group is in the process of revising the framework and working in the 
comments from the hearing. This work is expected to finish in summer 2008. Until 
now the use of the correct terminology for describing learning outcomes and associated 
concepts by the higher education institutions has been optional, but with the new 
national qualifications framework this is expected to become compulsory. 


5 The first Danish qualifications framework for higher education can be found at the following link: http://www.ciriusonline.dk/ 
Admin/Public/Download.aspx?file=files/filer/bologna/danish_qf_report_2003.pdf. 


Learning outcomes in the Danish higher education institutions 

EVA selected five higher education institutions to participate in the NOQA 
questionnaire on learning outcomes: 

• The Faculty of Humanities, University of Aarhus (referred to as HUM-AU). 

• Roskilde University (referred to as RUC). 

• The Danish School of Design (referred to as DKDS). 

• Roskilde Business College (referred to as RBC). 

• The School of Nursing in Silkeborg (referred to as SNS). 

These are, five very different institutions, covering the different programme levels in 
Denmark. The institutions have different experiences with learning outcomes, but all 
of them have been working with learning outcomes. 

The information has been gathered through telephone interview with the people 
responsible for learning outcomes at the five institutions. All of them have forwarded 
supplementary information and some of the institutions have also answered a few 
additional questions. 

INTRODUCTION OF LEARNING OUTCOMES 

HUM-AU, DKDS and SNS have introduced descriptions of learning outcomes in 
all their programme regulations. RUC has introduced learning outcomes in the 
programme regulations of their bachelor programmes, but not yet to their master 
programmes. RBC has started formulating learning outcomes for the Degree in 
Marketing Management and is expecting to finish this process in summer 2008. The 
teachers have been key actors in the process of formulating learning outcomes at all 
institutions. The students have also been involved at all institutions, except at RBC. 
Representatives from the labour market have generally been involved to a lesser degree. 
However, at SNS the clinical collaborators to the programme have been involved in 
the implementation, at RUC the careers guidance counsellor has directed the project 
of describing learning outcomes; and at HUM-AU a labour market advisory board 
has discussed learning outcomes at a very general level. Only HUM-AU involved the 
external examiners. 

DEFINING LEARNING OUTCOMES 

All institutions define learning outcomes by the competences the students are expected 
to have when graduating. Only DKDS and SNS have used the exact concepts of the 
new proposed second generation national qualifications framework (i.e. knowledge, 
skills, competences) in the descriptions of learning outcomes. All institutions have been 
inspired by the old or the new national qualification framework. HUM-AU and RUC 
has developed new terminologies for learning outcomes, which they consider exact and 
usable than the existing concepts of the national qualification framework. RBC has 
formulated six categories which they define as complementary to the concepts of the 
national qualification framework. These are to a greater extent than the concepts of the 
national qualification framework (2003) seen as useful in the day-to-day work of RBC. 
RUC and DKDS started formulating learning outcomes when learning outcomes had 
become part of the executive order for university programmes. HUM-AU, DKDS, SNS 
and RBS all took initiatives to formulate learning outcomes before this became part of 
the executive order of their programmes. 


USING LEARNING OUTCOMES 

All institutions find that the descriptions of learning outcomes are very useful to the 
institution. However RBC finds it difficult to assess the value of learning outcomes, 
since learning outcomes are not yet fully implemented. Many of the institutions 
incorporate learning outcomes in the programme regulations, to define examination 
requirements, as part of course descriptions and in the evaluation of the teaching. 
HUM-AU also uses learning outcomes in the teaching, while teachers inform students 
of how the themes of the lectures reflect the learning outcomes of the course. RUC sees 
learning outcomes as a useful tool in career counselling, since learning outcomes can 
help a student to select a particular study or career. RBC and RUC emphasize that the 
process of describing learning outcomes has opened up useful educational discussions 
among the teachers. 

All institutions have introduced learning outcomes to the teachers at meetings. 

RUC and HUM-AU also issued the teachers with notes about the institutions new 
terminologies of learning including guidelines on how to use them. RUC and HUM-AU 
have introduced their learning outcomes to the students in the managing councils. At 
HUM-AU all students are continuously informed via the teaching. At SNS the student 
council inform all the students about learning outcomes. DKDS has informed the 
students, external examiners and the business community. All institutions are planning 
to follow up or improve the use of learning outcomes in the near future. The foci of the 
institutions are, however, very different. For instance Hum-AU is planning to evaluate 
the teachers’ application of concepts and categories related to learning outcomes. 

RUC is considering reforming the structure of the programmes, which will influence 
the learning outcomes of the programmes. SNS will evaluate the progression of its 
programme and the learning outcomes of the modules. 

EVA and learning outcomes 

EVA has taken part in the development of a national qualification framework and 
descriptions of learning outcomes since 2001. EVA has been part of both the Danish 
Bologna Follow-up group, which approved the national qualification framework, and a 
smaller writing-group that made the drafts discussed in the follow-up group. 

For EVA looking at learning outcomes in the evaluations has been a 'natural ' 
approach, since the process of developing a national qualification framework started 
in 2001. EVA’s view on the use of learning outcomes in evaluations is to include them 
whenever ithey are seen as relevant for the focus of the evaluation. 

In 2007-08 EVA has been developing a new set of accreditation criteria, focusing 
on quality and relevance. The new set of criteria has been tested at 14 programmes 
in the college sector. Learning outcomes will be an integrated part of the new set of 
accreditation criteria. The criteria have not yet been made public but in the latest 
version from EVA (handed over to the ministry of education which will incorporate 
them into a departmental order this summer) one of the criteria consider; 

• whether the description of learning outcomes (programme level)is in line with 
the degree description in the national qualification framework (demands for 
respective level/degree) and the descriptions of learning outcomes at module level 
are coherent with the learning outcome descriptions at the programme level; 

• whether tests and examinations secure a adequate illustration of whether or not 
the student has attained the expected learning outcomes; 


• whether the graduates attain the expected learning outcomes to a satisfactory 
level. 

In the forthcoming accreditation criteria for academy profession programmes and 
professional bachelor programmes learning outcomes form an integrated part. The use 
of learning outcomes in evaluations is therefore set to increase. 


The Finnish higher education system 

The Finnish higher education system comprises universities and polytechnics, in 
which the admission requirement is a secondary general or vocational diploma. There 
are currently 20 universities in 11 different cities and towns in Finland. They are all 
state-owned and mostly financed from the state budget. Their operations are based 
on the principles of freedom of education and research, and university autonomy, 
guaranteed by legislation. Universities offer bachelor and master degrees in addition to 
postgraduate licentiate and doctoral degrees. 

According to the Government Programme, structural development will continue 
in both sectors of higher education in Finland. The main lines were set out in the 
Development Plan for Education and Research for 2008-2012 adopted in December 
2007. It is based on the dual model (of both universities and polytechnics) and on 
stronger regional and domain-specific profiles for the institutions. The aim is to 
enhance the quality of education and research, in addition to securing access to 
competent work force. 
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Fig 2 Illustration of the Finnish higher education system 












National qualifications framework in Finland 

The Ministry of Education has produced a preliminary report on the issue of a 
qualifications framework: Description of Finnish higher education qualifications 
in 2005. This report does not encompass for all the elements of a proper national 
framework, for example, learning outcomes. 

Currently the Ministry is in the process of producing the final version of a 
Development Plan for Education and Research for 2007-2012. The draft of this plan 
states that a national qualification framework will be produced by 2010. Producing a 
comprehensive framework is considered a fairly arduous task, and at the moment the 
Ministry is weighing up reasons for and against a detailed framework. The Ministry 
will most likely form a working group to create a new national framework, or enhance 
the old report in 2008-2009. 

Finnish higher education institutions adopted the two-tier degree structure 
in all study fields in 2005. The institutions have also created so-called core-skills 
analyses, which describe and assess the most essential topics in each course and study 
programme. These descriptions have not been used in FINHEEC evaluations, but they 
may well play a role in the planned Centres of Excellence in University Education- 
evaluations. 

The Finnish Polytechnics Rectors Conference launched a project in 2004 on 
the participation of the polytechnics in the European Higher Education Area. The 
background for the competencies was set by the European Qualification Framework 
and the results of the Tuning project. In the second phase of this project the 
polytechnics concentrated on creating descriptions of learning outcomes for both 
subject-specific competencies of the study programme and generic competences of all 
polytechnic graduates. Almost all fields of study were included. The objective was to 
create a comprehensive collection of competence descriptions that would be easy to 
comprehend, could be remembered and which actually could steer and direct teaching 
and learning processes in practice. The competence descriptions for all polytechnics 
were formed in a series of subject-specific workshops and seminars during 2005-2006. 
The project steering group made a number of recommendations for defining the 
competencies in subject specific groups in the winter of 2006: 

• The aim is to describe the study-programme related abilities of a graduate through 
3-6 competencies. 

• Competencies must be separable from each other, and their evaluation must be 
possible as separate entities. 

• The form of the competence descriptions should be made in co-operation with 
teachers, students and external stakeholders. 

The group formed six generic competence descriptions: learning, ethical, 
communicative and social, development, organisational and societal; and 
internationalisation competence. 

The Polytechnics Rectors’ conference produced a recommendation for using the 
matrix model for describing curriculum and acquirement of competencies. Almost 
50 % of Finnish polytechnics have adopted this method of curriculum description. 

The model shows which courses and modules are related to which generic and 
subject specific competencies. In May 2007, 85% of polytechnics had defined their 
competencies according to the national recommendations, and the remaining 15% had 
chosen some other method of competency classification. The produced descriptions 


have been adopted by the majority of institutions with only fairly limited adjustments. 
Measurement of the way in which graduates are actually attaining the described 
competences has only begun. Polytechnics are currently creating indicators for 
evaluating learning outcomes. 

Learning outcomes in the Finnish higher education institutions 

The standard NOQA questionnaire on learning outcomes and evaluation was sent to 
three Finnish institutions of higher education: Laurea and Jyvaskyla polytechnics 6 and 
the Sibelius Academy (SibA). Laurea is the fourth largest polytechnic in Finland and 
operates in the Helsinki metropolitan area. It has 8000 students in six fields of study. 
Jyvaskyla polytechnic (JAMK) operates in the Central region of Finland and is roughly 
the same size as Laurea. Sibelius Academy is the only music university in Finland. It has 
about 1700 students and 183 teachers, most of work in Helsinki. 

INTRODUCTION OF LEARNING OUTCOMES 

All institutions have introduced learning outcomes descriptions for degree 
programmes. Laurea tied the creation of learning outcomes closely to curriculum 
formation. Both polytechnics, Laurea and JAMK formulated the learning outcomes 
in congruence with the demands of working life and the descriptions of knowledge 
areas for expertise. The descriptions were discussed with teachers, students and other 
stakeholders. In Laurea, stakeholders were involved through interviews, meetings and 
curriculum seminars. In SibA, departments were actively involved in the creation of 
descriptions. This work was co-ordinated by the Teaching and Research Council, which 
includes the representatives of professors, senior assistants and students. 

DEFINING LEARNING OUTCOMES 

All three institutions define learning outcomes in a fairly similar way. Learning 
outcomes are understood to mean the knowledge, skills and competences that 
the students must and will possess in order to graduate or pass a study unit or a 
module. JAMK considers learning outcomes of an individual course as the minimum 
competences necessary to pass, and learning outcomes of the degree as a description 
of the skills and competences which the graduate will have after completing studies. 
Both polytechnics used the European qualifications framework and the draft of the 
National qualifications framework. Laurea created its own learning outcomes and 
JAMK employed learning outcomes created by the Rectors’ Conference project. SibA 
is using a version of the European qualifications framework and the music version of 
the Dublin Descriptors. The initiative for learning outcomes creation arose through the 
European cooperation in the Polifonia-network, which SibA is very much involved in. 
The polytechnics took the initiative themselves in order to implement the ideas of the 
Bologna process. In JAMK, predictions of the demands of working life also played an 
important role. 


6 Most Finnish polytechnics have chosen to translate ammattikorkeakoulu to University of Applied Sciences. The Ministry of 
Education has chosen to continue using the term polytechnic and therefore the FINHEEC continues to use it as well. 


USING LEARNING OUTCOMES 

Learning outcomes are used in polytechnics for the tutoring and guidance process 
of students, recognition of competences at the beginning of studies and the 
communication of degree programme profiles. Furthermore, Laurea and JAMK use 
the learning outcomes in the evaluation of students. The former also uses the learning 
outcomes descriptions as a basic part of the feedback process, and evaluates students’ 
generic competences with a qualitative portfolio-assessment, while subject-specific 
competences are evaluated on a scale of 1-5. SibA utilises learning outcomes for 
curriculum creation by probing music industry expectations and demand for future 
graduates. All of the institutions in question find learning outcomes essential and 
extremely effective, since they affect student recruitment, recognition of competences, 
evaluation, co-operation between study programmes and stakeholders. The learning 
outcomes also create the basis for curriculum creation. 

Learning outcomes descriptions and the use of learning outcomes has been 
introduced to relevant parties in all of the higher education institutions in question. 

The polytechnics have had meetings with student associations and student tutors have 
been trained to ‘spread the word’ concerning learning outcomes. Also, personnel and 
stakeholders have been involved in the process. In SibA, both teachers and students 
have been informed about the process. Involvement was secured by their participation 
in the decision-making. 

The follow-up work on learning outcomes seems to be closely tied to the curriculum 
evaluation and readjustment processes in the institutions. The learning outcomes are 
therefore continuously improved and simplified in order to guarantee their ease of 
usage and to limit different interpretations within the institutions. 

Overall, it seems that learning outcomes have been introduced and implemented 
fairly thoroughly to the three institutions in question. The polytechnics can be 
considered quite representative of the whole Finnish polytechnic population. However, 
the Sibelius Academy can perhaps be considered a pioneer in the Finnish university 
sector in this regard and unfortunately cannot be considered a representative example 
of the system as a whole. It does provide other universities with a positive example 
of the benefits of utilising learning outcomes. Since crossing sector lines seems to be 
somewhat difficult, especially for the universities, the Sibelius Academy may provide a 
very valuable benchmark to other universities in the near future. 

FINHEEC and learning outcomes 

The Finnish Higher Education Evaluation Council does not yet have an official view 
on the use of learning outcomes in evaluations. The agency had a representative in the 
working group which produced the preliminary report: “Qualifications Framework. 
Description of Finnish higher education qualifications”. It is possible that FINHEEC 
will take part in the formation of the national qualifications framework. Since the 
use of learning outcomes is fairly new in Finnish higher education, and almost non- 
existent in the university sector, the FINHEEC has not so far used learning outcomes in 
evaluation criteria. The Board of Professional Courses, which worked under FINHEEC 
until 2007, received some applications with descriptions of learning outcomes for 
professional courses. FINHEEC launched two evaluations on Centres of Excellence 
in Education on polytechnic and university sectors in 2008, which should outline the 
current situation in the use of learning outcomes in Finnish higher education. 


The bulk of FINHEEC 's evaluations consist of audits of quality assurance systems 
of higher education institutions. As the institutions create and adopt learning 
outcomes, they may be used more often as a tool for assessing student and therefore 
also organisation performance. Then, the learning outcomes would become an integral 
part of the higher education institution’s quality assurance systems and a target for 
FINHEEC 's audits. Once the learning outcomes are utilised system-wide, they could be 
included in the evaluation criteria of Centres of Excellence evaluations in both sectors. 

The Finnish Higher Education Evaluation Council publishes an action plan for action 
for its two or four year terms. The draft plan for 2008-2009 includes a proposal for a 
thematic evaluation of the use of learning outcomes in Finnish higher education. The 
phenomenon is, therefore, extremely topical in Finland at the moment, and FINHEEC 
intends to be closely connected to developments in the future. 


The Icelandic higher education 
system 

The system of higher education in Iceland is divided into three cycles that comply 
roughly to the Bologna- process. With the new Higher Education Act of 2006, 
the system was adapted to the Bologna process, including an obligatory Diploma 
Supplement, the ECTS-system and the introduction of the Icelandic national 
qualification framework (INQF). 
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Fig 3 Illustration of the Icelandic higher education system 
The organisation of quality assurance in Iceland 

In the absence of a formal quality assurance agency in Iceland, the organisation of 
external evaluations of tertiary education comes under the auspices of the Ministry 
of Education, Science and Culture. However, the ministry does not directly conduct 
any external evaluations. The results of the questionnaires to the agencies do therefore 






reflect the role of national qualification framework and learning outcomes in the direct 
organisation of quality assurance in Iceland. 

The Icelandic national qualification system 

The INQF is a systematic description of learning outcomes and the structure of the 
degrees that are awarded at higher education institutions in Iceland. The design of the 
INQF is based on the degree structure, stipulated by the Bologna process. In developing 
the INQF, a consultation with the major stakeholders of the higher education system 
was conducted. Systems from other countries such as the UK and the Nordic countries, 
as well as the Dublin descriptors, were discussed. As a result of this, the INQF is not 
just an adopted, but an adapted framework suited to the structure of a smaller nations’ 
education system. 

According to article 3 of the Higher Education Act of 2006, all higher education 
institutions had to undergo an accreditation before the end of June 2008. One of the 
main reasons for the accreditation procedure was to ensure that all the existing higher 
education institutions were adopting the national qualification framework, in order to 
ensure the rights of students as well as guarantee that the Icelandic system of higher 
education was adopting the standards of the European Higher Education Area. 

The qualifications requirements for the accreditation, concerned with the following 
aspects: 

a. role and objectives of higher education institutions 

b. administration 

c. organisation of teaching and research 

d. personnel qualification requirements 

e. admission requirements and student rights and duties 

f. working conditions for teachers and students as well as support structures 

g. internal quality management system 

h. description of study according to learning outcomes 

i. finances 

The regulations concerning the quality assurance of the future are still in the 
making. However, the aim of the system as a whole, the accreditation process, the 
obligatory internal quality assurance systems of the higher education institutions, the 
external quality assurance system based on external evaluations and the performance 
contract between the state and the institutions is being built around the core of the 
system, i.e. the Icelandic national qualification framework. 

Learning outcomes in higher education institutions in Iceland 

The standard NOQA questionnaire on learning outcomes was sent to three Icelandic 
higher education institutions: The University of Iceland (UI) 7 , Reykjavik University 
(RU) and the University of Akureyri (UA). The University of Iceland is the largest in 
Iceland with 9,000 students, six fields of study and 11 departments. The University 
of Reykjavik is a private, government dependent institution with 2,800 students, two 


7 According to law no. 63/2006 all higher education institutions in Iceland are entitled to call them selves “Haskoli” (Hogskola/ 
Hojskole) as the Icelandic language has no word for “University/Universitet” and the word Haskoli is normally translated as 
University. However, according to the same law, the Minister of Education decides, in accordance with a forthcoming regulation, 
what Icelandic HEIs are allowed to call themselves in foreign languages. 


fields of study and four departments. The University of Akureyri is located in the North 
of Iceland and has 1,500 students, three fields of study and four departments. 

All three higher education institutions have described learning outcomes for all 
programmes. UI is in the process of describing learning outcomes for all courses, but 
UA and RU have described the learning outcomes for all study courses. 

At the smaller universities (RU and UA) the learning outcomes were formulated 
by deans, teachers and experts within each school/faculty, seemingly with little 
participation of students. At UI, the method of describing learning outcomes was 
different from one faculty to another. Some faculties started by formulating learning 
outcomes for individual courses and described learning outcomes for the programmes 
in accordance with the learning outcomes for the courses. Teachers of the programmes 
in question were always involved and sometimes stakeholders were contacted. Student 
participation in the process varied. 

DEFINING LEARNING OUTCOMES 

According to UI, the learning outcomes are defined as a statement of what a learner 
is expected to know, understand and/or be able to demonstrate after the completion 
of learning. Learning outcomes are sets of competences, expressing what the student 
will know, understand or be able to do after completion of the given programme or 
individual courses. Learning outcomes specify the minimum requirements for the 
award of credits or for graduation from a programme. RU and UA gave similar answers, 
although RU added that the learning outcomes are important to a student and to each 
core because they make them aware of certain goals/objectives that students and 
institutions have to reach. 

USING LEARNING OUTCOMES 

The INQF was made by the institutions in cooperation with the Ministry of Education. 
All the institutions used the national qualification framework as a model for the 
description of learning outcomes. 

UI claimed that Bologna-promoters had started the discussion about writing 
learning outcomes for study programmes at the institution. The decisive factor was 
in the end the new Higher Education Act of 2006 that enforced the institution to 
describe learning outcomes. Prior to the act of 2006, RU had begun describing learning 
outcomes for courses, based on the ECTS-label. UA started the ECTS-label and Diploma 
Supplement process in 2004. 

All the institutions claimed that the learning outcomes were published in the 
University catalogue of each institution so students, stakeholders and the labour market 
should be able to figure out which competences students should have at the completion 
of a given programme or course. While UI only publishes the learning outcomes in 
the public university catalogue, RU publishes them on the institution’s website and in 
e-mails to students. UA does the same but has also held workshops for teachers and 
other staff on learning outcomes. 

All the institutions claimed that the learning outcomes were very useful for the 
institutions especially for the reviews of courses and programmes. UA found that 
the description of learning outcomes had affected the teaching methods in the 
programmes, as the teachers have to utilise different types of methods and assessments 
to fulfil different types of learning outcomes. 


Learning outcomes will in the future be used in the internal quality assurance and in 
the revision of programmes e.g. by presenting a matrix with learning outcomes for the 
programme in question on one side and course descriptions on the other. This would 
be to ensure that all learning outcomes are represented in the courses and to prevent 
overlap. All institutions in question have made plans to revise their learning outcomes 
regularly, and in that context, to consult stakeholders. At RU, the review will be 
undertaken every two years by the deans and teachers. UA is currently concentrating 
on the connections between learning outcomes, teaching and assessment. 

The Ministry of Education, Science and Culture and learning outcomes 

As stated above, the learning outcomes were one of the criteria that the higher 
education institutions had to fulfil to be accredited. In the coming three year plan 
of the Ministry for quality assurance in higher education, the learning outcomes of 
programmes will make up one of the core criteria in external evaluations, both at 
programme level and institutional level. 


The Norwegian higher education 
system 

Norway has in place a structure of higher education that complies with the Bologna 
process. In 2003, the Bachelor-Master-PhD structure was implemented in almost all 
higher education study programmes, students were credited 60 ECTS for one year of 
academic studies, and the Diploma Supplement was also implemented for all of the 
programmes. Most of the Norwegian higher education institutions have a description 
of the curriculum for their study programmes. Included in these are descriptions of the 
objectives. These objectives shall state the knowledge, skills and attitudes that students 
should have acquired upon completion of the study programme. 

Norway has three categories of institutions accredited for higher education: 
university college, specialised university and university. The institution category 
determines the authority which the institution has to establish new study programmes 
and courses without having to apply to NOKUT for initial accreditation. 
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Fig 4 Illustration of the Norwegian higher education system 







Situation related to national qualification framework and learning outcomes 

In the spring of 2007 the Ministry of Education and Research sent to a hearing a 
proposal for a national qualification framework to the institutions of higher education 
and other stakeholders. The content of the framework is close to the descriptions of 
learning outcomes that are presented in the European Qualification Framework and 
the Dublin Descriptors. 

On the basis of the comments from the institutions and stakeholders, it is expected 
that the Ministry will decide upon a national qualification framework. At the moment, 
there is no information about when this might be expected. Several Norwegian higher 
education institutions are presently working to develop learning outcomes. Until 
the national qualification framework is decided upon, the learning outcomes are 
yet not described according to a level, but to an academic tradition. Before 2010, the 
institutions will have to adjust all their study programmes and diplomas to ensure that 
they describe learning outcomes. 

Learning outcomes in the higher education institutions 

NOKUT selected three institutions to participate in the NOQA questionnaire survey of 
learning outcomes. All these institutions claim they have begun a process of describing 
learning outcomes for some or all study programmes and courses. These are: University 
of Oslo (UiO), University of Tromso (UiT) and The Norwegian School of Theology 
(MF). 

The University of Oslo is Norway’s largest higher education institution. Today, UiO 
has approximately 30,000 students and 4,600 employees. Located at latitude of nearly 
70 degrees north, the University of Tromso is the world’s northernmost university. UiT 
is a relatively small university, with around 6,000 students and around 1,800 staff. MF 
is a private specialized university in the field of theology and religious studies and the 
largest theological institution in Norway. 

INTRODUCTION OF LEARNING OUTCOMES 

MF - Norwegian School of Theology has introduced the concept of learning outcome 
in some programmes and courses. At the moment the institution is working on 
introducing learning outcomes in all courses. At the University of Tromso the Faculty 
of social science has started a project to revise all programme and course descriptions. 
The project was introduced through a seminar for the administrative staff and relevant 
teachers. The University of Oslo introduced learning outcomes through a seminar 
including a workshop for all relevant academic and administrative staff. 

DEFINING LEARNING OUTCOMES 

The University of Tromso uses the Dublin descriptors as a basis for formulating 
learning outcomes. Learning outcomes are defined as learning results. UiT defined 
the word ’’outcomes” as being more instrumental and commercial. The concepts could 
be understood as ’’effort gives outcomes”. Knowledge, skills and competence are not 
mutually exclusive categories; they assume each other’s presence. At the Faculty of 
social science they use an analytical distinction between knowledge and skills. The 
faculty wants the students to focus on using their knowledge and skills, and develop 
independence in their studies. This faculty has an internal working-group that have 
developed examples of descriptions of programmes and courses, the purpose of which 


is to create more unified descriptions and easier implementation for the departments. 
These examples are discussed and approved by the relevant bodies at the university. 

The departments observe the descriptions of the courses and programmes. 

MF has meetings every year with different stakeholders, where they discuss the 
need for specific learning outcomes. However, the stakeholders do not participate in 
formulating learning outcomes in the course descriptions. 

UiO has not drafted a formal definition of learning outcomes so far, but the 
institution has issued an instruction for publishing information about their 
programmes on the internet pages: 

Learning outcomes should: 

• describe the knowledge, skills and attitudes the students achieve during the 
programme; 

• describe academic- and other competence that the programme gives; 

• describe the objectives related to the teaching and assessment in the programme; 

• for a master programme: describe the objectives of the master theses and possible 
fields of deeper understanding. 

Some faculties have used the proposed national qualification framework as a basis for 
defining learning outcomes. Others are influenced by the Dublin descriptors, or the 
Danish national qualification framework from 2003. 

External stakeholders do not influence the description of learning outcomes at any 
institutions. 

USING LEARNING OUTCOMES 

All institutions find the descriptions of learning outcomes very useful. Since the 
project of implementing learning outcomes has not been completed by the institutions, 
experience is limited. The most valuable experience so far is the reflection on the 
subject and programme, and the discussions about content formulations, attitude and 
the teaching outcomes. 

The institutions use the learning outcomes to describe the objectives, qualifications 
and the competences the students are expected to have achieved upon graduation. UiT 
states that the aims of the descriptions are to provide student information. MF uses 
the learning outcomes to make the programmes more adequate for future professional 
careers. The purpose of the description of knowledge, skills, and the attitude associated 
with all courses and study programmes at the University of Oslo is to give the 
students information, so that they can make informed choices about their studies. All 
programmes should also describe the attained competence upon completion. Examples 
can be seen from the master programme in Peace and Conflict studies at: http://www. 
uio.no/studier/program/peace-master/om/hva-laerer-du.xml and the course description 
in Human Rights and Inclusion at http://www.uio.no/studier/emner/uv/isp/SNE4110/ 
index.xml . 

UiO uses the learning outcomes as part of the diploma, and has made plans to 
use them in the Diploma Supplement. Some of the teachers at UiO have begun using 
learning outcomes in their teaching. In its plan for strategic work UiO, has decided to 
use this period to prepare for better knowledge in the programmes and ensure that this 
strengthens the teaching and assessment of students. Learning outcomes form part of 
the quality assurance system at UiT and UiO. 


The medicine programme at UiO implemented learning outcomes in 1996. The 
students say that they understand the learning outcome as a form of tutoring and 
guidance to obtain the expected competence. They are better prepared for examination 
because they have more specific information about what is expected of them. Ten 
years after implementing the Reform of -96, the Medicine Faculty evaluated this 
study programme. The candidates made a self-evaluation of their personal skills. One 
of the main questions was whether candidates in the period between 1996 and 1999 
were better prepared to handle practical skills when they began practicing as medical 
doctors. The results did show that this group of students were better prepared when 
they started practicing as doctors, but advantages decreases during the first one or two 
years of practice. 

NOKUT and learning outcomes 

NOKUT took part in the Ministry’s working group, which made the proposal of a 
national qualification framework and descriptions of learning outcomes. NOKUT is 
taking part in the Bologna follow-up reference group. NOKUT have since 2003 issued 
regulations close to learning outcomes and undertaken evaluation on the basis of these 
regulations. In Norway, the learning outcomes are closely connected to a national 
qualification framework. A basic element in a national qualification framework is 
that the qualification should be outcome related and described in terms of learning 
outcomes rather than input factors. NOKUT undertakes evaluations such as: 

• audits (evaluation of a higher education institution’s system of quality assurance); 

• accreditation (initial accreditation of courses, programmes or institutions); 

• revision of previously granted accreditations (evaluations of previously granted 
accreditations); 

• general evaluation of specific types of educational provision or defined aspects of 
such. 

In the future, it is expected that NOKUT will undertake evaluations incorporating 
learning outcomes. It is expected that all these evaluations in some way must include 
this perspective. By 2010, NOKUT will have to revise its regulations and further 
develop its evaluation methods with a view to including learning outcomes in all 
evaluations. 


The Swedish higher education system 

In Sweden there are 36 state-owned universities and university colleges providing 
higher education. Of these, there are 14 universities with the right to offer third level 
education, i.e. doctoral level. There are also 13 private providers of higher education. 
Three of these, has the right to offer third level education (Chalmers University of 
Technology, Stockholm School of Economics and Jonkoping University). 

Universities and university colleges have the general right to award undergraduate 
degrees (Diploma and Bachelor). They also have the general right to award the l-year 
master (postgraduate level). The universities have the general right to award the 
2-year master as have the university colleges which are entitled to conduct research 
in the particular discipline domains in which they are permitted to award third level 
qualifications. Other university colleges must apply to the HSV to obtain the right to 
award a 2-year master in a subject/discipline. 

Since 1 July 2007 Sweden has adapted the qualification framework for higher 
education according to the Bologna declaration, i.e. first, second and third level 
education. 
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Fig 5 Illustration of the Swedish higher education system 
The Swedish national qualification framework 

Sweden has taken a legislative approach to the development of a national qualification 
framework. Recent amendments to the Higher Education Ordinance align the Swedish 









structure for degrees and programmes with the Bologna process. All qualifications are 
now defined in terms of learning outcomes, levels (using the first, second and third 
cycles specified in the Bologna process) and workload (using the European Credit 
System). Generic as well as specific (professional) qualifications are outlined. 

Based on the legislative changes, HSV was commissioned by the Swedish 
government to draft a proposal for a national qualification framework. The framework 
was to be developed in collaboration with higher education institutions and other 
relevant stakeholders. In essence, the task meant providing a description that was more 
user-friendly than the legislation itself. Neither the school system nor life-long learning 
structures were to be included. A proposal was presented to the Ministry in June 2007 
(http://www.hsv.se/d0wnl0ad/18.5b73fe55111705b51fd80005519/12-326-07_referensram. 
pdf). 

The proposal provides a presentation of the Swedish structure of degrees and 
programmes following the recent amendments to the Ordinance. The part describing 
the qualification system itself is relatively brief, as this is already described in detail 
by the Swedish law. The part describing admittance to higher education is described 
in more detail as being one of the few areas which are more centrally regulated. This 
proposal has been accepted by the Ministry, and they have commissioned HSV to 
carry out the final updating of the document and to make it publishable to different 
stakeholders, as well as to produce an English translation of it. This work is now in 
progress. 

Learning Outcomes in the higher education institutions 

In order to describe the introduction and implementation of learning outcomes in 
Sweden three institutions with different size and background are chosen; University of 
Gothenburg, Karolinska Institute (KI) and Malmo University College. 

All three higher education institutions have introduced learning outcomes in all 
courses and all study programmes. 

INTRODUCTION OF LEARNING OUTCOMES 

The common denominator for all three institutions was that they have organized 
workshops and seminars on the subject for teachers, education planners and students. 

As a part of the implementation of the Bologna process, the University of 
Gothenburg established an action plan in which the implementation of learning 
outcomes was referred to. A working group has elaborated guidelines for the 
departments on writing syllabi containing learning outcomes. 

All three institutions reported that learning outcomes are introduced to all internal 
stakeholders. Stakeholders outside the university have not been much involved in the 
process, but will be as the learning outcomes develop and the connection with the 
labour market increases. 

DEFINING LEARNING OUTCOMES 

In the Swedish higher education institutions there are slight differences between the 
declared definitions of the respondents, but they all agree that learning outcomes are 
statements that express what students are expected to know, understand, and be able to 
do at the end of a course or a study programme. 


The institutions have used different documents as guidelines to introducing and 
implementing learning outcomes. The University of Gothenburg and KI reported 
that besides the national qualification framework and the European Qualification 
Framework they have used the university’s own guidelines. Malmo has used the 
Examensordning i HF, EUA Bologna handbook and documents from several other 
European universities. 

USING LEARNING OUTCOMES 

The institutions have reported that they mainly use learning outcomes as performance 
indicators and as a tool in the evaluation of the courses and programmes. 

All three institutions attached a high value to learning outcomes. The most 
frequently stated merits of learning outcomes are: 

• that they provide a better understanding of what a course or programme offers the 
student in terms of skills, competences, etc; 

• how they relate to a certain labour market or further studies; 

• how they provide better information to the student about the curricula. 

Teachers and stakeholders are also mentioned as groups that will benefit from a well 
implemented set of learning outcomes. Learning outcomes offer a good basis for 
pedagogical development of the courses and programmes themselves; they strengthen 
the connection between content, examination and assessment. 

The University of Gothenburg stated that the concept of learning outcomes 
must become a known, everyday part of the planning of courses and programmes. 
Departments will gradually look over their syllabi and start developing the learning 
outcomes. Forthcoming seminars, for instance the ECA seminar in Zurich in 
September 2007, and actions planned together with our centre for Pedagogical 
Development and Interactive Learning (PIL), are ways of enhancing staff competence 
in this field and supporting the ongoing pedagogical development. Malmo is planning 
to organize seminars to further develop the thinking surrounding learning outcomes in 
relation to examination criteria. 

The Board of Education at KI has initiated an evaluation project in order to follow 
up changes in course design, assessment and requirements of knowledge due to the 
implementation of learning outcomes. 

HSV and learning outcomes 

Sweden has recently, 1 July 2007, adopted the Bologna system in higher education. 

HSV attached a high value to the Bologna process. Learning outcomes are an important 
and interesting part of this process that will have an impact on education itself. 

HSV can now observe that learning outcomes have been both adopted and adapted 
into the various higher education institution cycles for almost all courses and study 
programmes. Work has recently commenced to develop learning outcomes for degree 
curricula i.e. general qualifications at first, second and third levels; qualifications in 
the field of arts; and professional qualifications. Learning outcomes for both study 
programmes and degree curricula will be evaluated within the new cycle of the 
education evaluations. 

A proposal for a national qualification framework has recently been presented 
by HSV in cooperation with representatives from the Swedish higher education 
institutions. Representatives of stakeholder organisations have also given their 


comments on the document. Learning outcomes are closely related to the national 
qualification framework. The new evaluation cycle of education at institutions 
including learning outcomes of programmes and degree curricula, has recently begun, 
and information has not yet been presented. Expected knowledge, applied knowledge, 
communication skills and learning skills are described in course and programme 
syllabi. This documentation has always been an important basis in the evaluation of 
subject and programmes in the previous cycle. However, applying knowledge and 
employability in a wider perspective of learning outcomes has not been the focus of the 
previous evaluations. 


Summary: Learning outcomes in 
Nordic higher education institutions 

This section attempts to provide a summary of the current status regarding the use of 
learning outcomes in the selected higher education institutions. How they formulate, 
define, describe and use learning outcomes, and from where the inspiration came to 
start their work with them. 

Formulation of learning outcomes 

In all the institutions, the teachers have played a crucial role in the formulation of 
learning outcomes. Students have also been involved in the process of formulating 
learning outcomes, but generally to a lesser extent than the teachers. The methods 
of formulating learning outcomes vary among the participating higher education 
institutions. Some have held workshops and seminars on the subject, while others have 
had discussions. 

The involvement of external stakeholders also varies a great deal among the 
institutions. In some institutions external stakeholders have not been involved at 
all, while others have involved them in different ways. One institution has involved 
external stakeholders through interviews, meetings and curriculum seminars in order 
to secure or be able to produce descriptions of learning outcomes in accordance with 
the demands of working life, as well as descriptions of knowledge areas for expertise. 

At another institution, a labour market advisory board with 20 external members from 
the labour community have discussed the formulation of learning outcomes at a general 
level. The institution wanted to involve the labour market advisory board to a larger 
extent, but considering the workload related to the descriptions of learning outcomes 
as well as the tight timetable, they found such an involvement unrealistic. There is also 
an example where an institution has discussed the formulation of learning outcomes 
with selected members off a labour market advisory board in relation to the processes 
of setting up new programmes. 

Defining the concept of learning outcomes 

The definition of learning outcomes is fairly similar across the participating Nordic 
institutions. They all define learning outcomes as the qualifications and competences 
a student is expected to have at the completion of learning. But the taxonomic 
description of learning outcomes (the categories they use when describing learning 
outcomes) varies among the institutions. Some divide learning outcomes into three 
categories: knowledge, skills and competences. Others describe learning outcomes 
in terms of what the students are expected to know, understand and be able to 
demonstrate when graduating from a programme. Others define learning outcomes by 
the qualifications, competences, skills, attitude and knowledge of the graduates. 

There are institutions which have developed their own categories for describing 
learning outcomes. They see their terminology as more precise and usable than the 
existing concepts of learning outcomes. One of the institutions explained that they felt 
that the categories of learning outcomes were missing a dimension, since the category 
“attitude” was not included, and they chose to include “attitude” as a dimension. 


Another institution has formulated six categories which they define as complementary 
to the three categories in the national qualification framework. They regard the six 
categories as more useful in the day-to-day work of the institution. 

Sources of inspiration and the initiative to formulate learning outcomes 

In countries that have a national qualification framework, almost all participating 
institutions have used or been inspired by this framework in the process of formulating 
learning outcomes. Furthermore, many of them have used or been inspired by the 
European qualification framework. In addition to these two sources, the participating 
institutions mention that they have found inspiration in the Dublin descriptors, the EU 
Bologna Handbook, guidelines from other universities, the internet, participation in a 
pilot project, the taxonomy of Bloom and Biggs, and Luhmann’s systemic theory. 

The term of where the initiative to describe learning outcome have came from, 
the institutions can be divided into two groups. For some institutions, the initiative 
to formulate descriptions came from external sources. They mention a reform of the 
higher education system, the Bologna process and/or an executive order as external 
initiators. For other institutions, the initiative was taken internal by the institution 
themselves. 

Formulations and descriptions of learning outcomes 

There are several ways in which learning outcomes are used in Nordic higher 
education institutions. Learning outcomes are mainly used to provide information 
about education, course content, modules, programmes and degrees. The important 
aspect to note here is that the descriptions are not limited to the knowledge contents 
(i.e. what information the student is expected to learn), but in particular the skills and 
competences that the student possesses upon completing the study unit are paramount. 
The learning outcomes are used to communicate this information primarily to the 
students themselves, but also to employers and other stakeholders. Teachers also tend to 
use learning outcomes in the guidance process as a basis of teacher-student discussions 
to highlight the most important contents of courses, and in connection with career 
guidance. 

Related to the information value are the functions of the learning outcomes’ in 
examinations. Given that the learning outcomes outline the knowledge, skills and 
competencies to be gained, teachers use these descriptions as a basis for examination 
criteria and requirements. The outcomes are also useful in the creation and adjustment 
processes for curricula, where the staff can analyse the curriculum and check whether 
courses overlap in terms of learning outcomes. By using learning outcomes, courses and 
programmes can be made more suitable in terms of the demands of working life. 

Using learning outcomes 

The great majority of participating institutions found learning outcomes extremely 
useful. Some were more hesitant in their assessment, given that they had very recently 
introduced the outcomes or were in the process of doing so. Among the primary 
strengths stated was the encouragement to develop teaching methods, since teachers 
must ensure that the competencies described are in fact created. Also, the connection 
between content, examination and assessment was seen to be improving. Providing 
better information about courses and programmes was seen as important, as already 


stated above. The discussions between teachers and students on learning outcomes 
have also reportedly created better understanding of subject contents and expected 
attitudes. Some institutions even saw the introduction of learning outcomes as 
changing the very role of universities from traditional knowledge-creation facilities 
towards a more education-centred and quality-education-dominated orientation. The 
responses indicated few or reservations regarding the usefulness of learning outcomes. 

The responses also indicated that learning outcomes were being used to probe 
working-life expectations and future developments. It seems hardly surprising that, 
since learning outcomes are in many institutions only just beginning to be described 
and utilised, their function is mostly informative. Learning outcomes are in some 
institutions used to evaluate teaching performance and also as the basis of a student 
feedback mechanism, where students are asked to assess their education in relation 
to the learning outcomes. There is reason to expect more ambitious usage as a tool to 
assess and develop teaching and institutional performance in the future. For example, 
some of the institutions mentioned intended to use learning outcomes as a part of their 
quality assurance system. However, one would expect to see more ambitious usage as a 
tool to assess and develop teaching and institutional performance in the future. 

Introduction of learning outcomes to students, teachers and other 
stakeholders 

A great variety of means have been used to introduce learning outcomes to students, 
teachers and other stakeholders. Most commonly, e-mails, information packages on 
websites and seminars have been utilised. Student representatives appear to have 
been most involved and in some cases specially trained to spread the word among the 
student population. Given that the learning outcomes have fairly important information 
value to employers and external stakeholders, it may seem surprising that these groups 
haven’t so far been very involved or well informed about learning outcomes. External 
stakeholder involvement seems easiest for programmes and faculties that have an 
existing close relationship with local employers, and where graduates a have fairly good 
idea of where they will most likely be employed. The development of a closer external 
stakeholder involvement is considered desirable, since learning outcomes would appear 
to present an easily understandable basis for discussions between students, teachers 
and employers. 

Following up the descriptions of learning outcomes 

Most institutions carried to have plans for following up and updating their learning 
outcomes, or at least were planning to create plans for continued improvement. The 
responses, however, failed to describe in detail which measures will be undertaken. It 
appears that learning outcomes have a naturally close relationship with curricula and 
any adjustments will in most institutions be made within the same process as curricula 
are updated. 


Learning outcomes in the Nordic 
quality assurance agencies 

The following section provides a brief overview of the kinds of experience the Nordic 
quality assurance agencies have of working with qualification frameworks and learning 
outcomes in connection with external evaluation. Each agency has presented working 
papers on their experience, and this section is based on these papers. The country 
reports show that the Nordic countries are at different stages of implementing learning 
outcomes. On one hand, Iceland has introduced learning outcomes in all higher 
education, while on the other hand Finland has only quite recently begun the work 
on a proposal for how to use a qualification framework. To sum up, the situation is as 
follows: 

National qualification framework 

In Denmark, a qualification framework for higher education was in place in 2003. A 
new qualification framework was proposed in 2007, and descriptions of competence 
goals and learning outcomes are an important element in both frameworks. In 
Finland, the Ministry is in the process of producing a Development Plan for Education 
and Research for 2007 - 2012. It is stated in the draft of this plan that a national 
qualification framework will be produced by 2010. It has not yet been decided how the 
framework should comply with learning outcomes. In Iceland, all higher education 
institutions are obligated to have a clear description of learning outcomes in study 
programmes in accordance with the national qualification framework. In Norway, 
the Ministry of Education in 2007 presented a proposal for a national qualification 
framework in which descriptions of learning outcomes are a core element. On the basis 
of comments from the hearing, it is expected that the ministry will soon decide upon 
a national qualification framework. Since July 2007, Sweden has through legislation 
adapted its qualification framework for higher education so that it complies with the 
Bologna declaration. All qualifications in Sweden are defined in terms of learning 
outcomes, levels and workload. A proposal for a national qualification framework was 
presented to the Ministry in June 2007. It is expected that this framework will be 
officially adopted during spring 2009. 

All the agencies state that they have been represented in working groups during the 
preparation and development of their national qualification frameworks. 

Looking at learning outcomes 

Even though the work on introducing qualification frameworks and the associated 
mandatory learning outcomes, are at different stages in the different countries, 
some of the agencies have experience of using learning outcomes in their evaluation 
work. NOKUT and EVA are the agencies that have in a systematic way used learning 
objectives as an important element in their regular evaluations or pilot-projects. The 
methods used by these two agencies are described in part D. 

The Ministry of Education, Science and Culture in Iceland applied the national 
qualification framework as a basis for the criteria for learning outcomes in full scale 


accreditation of their higher educational sector that was completed in spring 2008, and 
this intend to continue to apply this in approach future external evaluation. 

The other agencies have so far little experience applying learning outcomes in their 
evaluations. FINHEEC has not used learning outcomes as a basis for evaluation criteria. 
Expected knowledge, applied knowledge, communication skills and learning skills 
are described in course and programme syllabi, and this documentation has always 
been an important basis for HSV’s evaluation of subjects and programmes in previous 
evaluation cycles. 

All agencies expect the introduction of learning outcomes to bring some changes in 
the way they are carrying out their evaluations. EVA uses a range of different methods, 
including programme accreditation, audit, benchmarking and evaluation. For coming 
programme accreditations, learning outcomes will be an important element of the 
evaluation, and in other kinds of evaluations they will be included where relevant. 
FINHEEC expects learning outcomes to become an integral part of quality assurance 
systems for the higher education institutions, and an aim to use them in the agency’s 
audits. They might also be included in the evaluation criteria for Centres of Excellence 
evaluations. HSV will include learning outcomes in their programme evaluations. The 
ministry in Iceland is currently working on a three year plan for external evaluations 
within higher education, and will take into consideration the learning outcomes of 
study programmes and courses and investigate whether they are in compliance with 
the national qualification framework. NOKUT undertakes audits, initial accreditation 
of programmes, accreditation of institutions, revisions to accredited programmes and 
general evaluations, and expect that all these evaluations will to some extent include 
the learning outcome perspective by 2010. 

The various legislation in the Nordic countries and the internal decisions of the 
quality assurance agencies show that the Nordic countries have different approaches to 
the interpretation and implementation of learning outcomes. EVA has the opportunity 
to determine for each evaluation whether it is relevant to include learning outcomes. In 
Norway the content of an evaluation is governed by the regulations. 


PART C. Agency experiences of 
evaluating learning outcomes in study 
programmes 

In this part the experiences of two agencies in evaluating using learning outcomes is 
presented. 

As part of the Bologna process, learning outcomes are to be implemented in all 
study programmes by 2010. The Nordic quality assurance agencies have, therefore, 
limited experience with evaluating based on learning outcomes. Denmark and Norway 
have since 2003 carried out certain evaluations based on learning outcomes of study 
programmes. 

In Denmark, EVA has undertaken some pilot-projects evaluating learning outcomes 
and the competences of graduates. In Norway, the institutions provide descriptions 
of programme objectives in terms of knowledge, skills, attitude and programme 
completion competence. 


EVA’s experience of evaluating 
learning outcomes 

EVA has incorporated a focus on competences and learning outcomes in several of 
their evaluations. In one of them, “German studies at the Universities” from 2003, part 
of the aim was to examine the quality of the German studies as expressed in the final 
competences of the graduates. But learning outcomes are not always included in EVA’s 
evaluations. It depends on whether or not it is appropriate or relevant to the focus of the 
evaluation. 

Developing a new set of accreditation criteria 

EVA and the Ministry of Education are in the process of developing a new set of 
accreditation criteria focusing on quality and relevance 8 . These are expected to be 
submitted to a hearing in June 2008. The accreditation criteria are an important part 
of the new accreditation system introduced in Denmark in spring 2007. According 
to the Accreditation Agency for Higher Education Act, no 294 of 27 March 2007, all 
higher education study programmes, and programmes in the parallel system for lifelong 
learning, are to be accredited and re-accredited every 5 th year (approximately) on the 
basis of the criteria 9 . 

To make sure that the new set of criteria is relevant and adequate, and to allow for 
adjustment to the criteria based on real-life experience, EVA has carried out a pilot test 
on 14 programmes during the summer of 2007. 

The pilot test 

Twenty four criteria were tested in the pilot test. The criteria were organised within 
four overall themes: 

• relevance and societal demand; 

• (sources of) educational knowledge; 

• content and organisation; 

• results and effect. 

The criteria dealt with a range of issues including intended and actual learning 
outcomes of the study programme and its relevance for the labour market, integration 
and transformation of trends from the professional field, research and development into 
educational knowledge, initiatives to prevent drop-out of students, etc. 

Each programme was assigned the task of describing and documenting how and 
to what degree it complied with the criteria and to comment on the criteria and 


8 At the moment, the criteria are being worked into an executive order which is expected to be sent to hearing in June this year. The 
executive order is expected to become effective this summer and to form the basis of the first accreditations in autumn 2008 

9 The law establishing the new accreditation system in Denmark outlines a number of important aspects of the system: Firstly, it 
specifies that one accreditation council is to be responsible for all accreditation decisions with respect to the various sectors of 
higher education in Denmark, but that different operators can undertake the specific accreditation assessments. It is stipulated 
that a secretariat is to form part of the new accreditation institution, and that this secretariat should be a prime operator within 
the university sector. Further, the law indicates that the ministries responsible for various sectors of the education system are 
ultimately responsible for deciding on the criteria pertinent to “their” study programmes. There is, in other words, room for 
several sets of criteria, but only one council. In order to safeguard transparency and comparability, it was, however, an expressed 
desire, that differentiation between sets of criteria should only take place in cases of fundamental differences between the sectors 
at hand (e.g. between academic education and professional/vocational education). 


the documentation possibilities. Once the material was handed in, EVA and an 
expert panel 10 discussed the material, made an initial assessment, and discussed the 
programme viewpoints on the criteria. Then a site visit was conducted, again with 
the double aim of, on one hand, investigate criteria compliance and supplementing 
documentation and, on the other obtaining more in-depth comments on the criteria 
and the working process. 

On the basis of the site visit, the initial assessment was revised, and any problems, 
need for clarification of each of the criteria, etc. were discussed. 

Looking at the achievement of learning outcomes in an accreditation 

One of the criteria in the pilot test looked into the achievement of learning outcomes. 
The criterion was formulated as follows: Taken together, the graduates of the programme 
achieve the intended learning outcomes to a satisfactory extent. 

The criterion should examine/check whether or not the graduates achieve the 
knowledge, skills and competences promised in the descriptions of the intended 
learning outcomes. The programmes were requested to describe and document 
their overall assessment of the learning outcomes of the graduates, the grades for 
the previous three graduating classes and how graduates (and employers) assess the 
achievement of the learning outcomes. 

At the site visit, EVA asked both graduates and employers about the achievement of 
the learning outcomes. But it was not easy to get valid answers to our questions in the 
interviews. Many of the graduates and the employers were not sufficiently aware of the 
learning outcomes of the programme. Other graduates had difficulties in separating 
knowledge, skills or competences achieved in the programme from knowledge, skills or 
competences achieved at the workplace - especially if they had graduated a year ago. 

The pilot test showed us that the use of interviews with graduates and employers 
was not suitable for assessing the achievement of the learning outcomes. Instead, EVA 
has chosen to look at the grades of the graduates and the number of students who not 
pass. EVA will still ask the graduates and the employers about the achievement of the 
learning outcomes at the site visits, but this will not be decisive to the assessment of the 
criteria. 

When looking at marks in relation to the criterion of achieving learning outcomes 
EVA expects that the assessment will be based on the average marks of graduates from 
the last three years. There will not be a fixed or absolute minimum requirement for 
the level of grades, but it will be seen as a signal of potential problems if the grades are 
below a certain level. 

In Denmark, grading in higher education is absolute and not relative. You measure 
the extent to which the goals or learning outcomes are achieved. You do not give grades 
relatively in comparison with the performance of other students. By giving grades in 
absolute terms, it is possible to compare marks given in different years. A grade from 
one year has the same value the following year, as it has been given in relation to how 
well you have achieved the goals or learning outcomes of the programme. 

EVA is aware that there are potential challenges regarding the use of grades as 
indicators of the achievement of learning outcomes. In a report from the Norwegian 


10 For each programme a specific expert panel was set up, consisting of three members: one with a programme content-oriented 
profile, one with an employer-profile, and one with pedagogical expertise. 


NIFU-STEP (NIFU-STEP report 40/2007) the use of grades is questioned, in the 
sense of what do examinations measure? There is a risk that traditional examinations 
only measure the student’s ability to memorise and not to understand. It can also 
be discussed whether a good mark reflects a good professional understanding. Also, 
because professionalism is assessed differently from one programme to another, marks 
and achievement of learning outcomes are difficult to compare between subjects. 

But despite these potential challenges, EVA still thinks that marks are the best 
alternative to measuring the achievement of learning outcomes. 

Learning outcomes in the new Danish accreditation criteria 

Learning outcomes will be an integrated part of the new set of accreditation criteria. 
The criteria have not yet been made public but in the latest draft version one of the 
criteria will consider whether: 

• the description of learning outcomes (at programme level) is in line with 
the degree description in the national qualification framework (demands for 
respective level/degree), and the descriptions of learning outcomes at module level 
are coherent with the learning outcome descriptions at the programme level. 

Another criterion will consider whether: 

• the tests and examinations are providing an adequate indication whether or not 
the student has reached/gained the expected learning outcomes. 

And a third criterion will consider whether: 

• the graduates are gaining the expected learning outcomes at a satisfactory level. 


NOKUT’s experience of evaluating 
learning outcomes 

The Norwegian Ministry of Education and Research is working on developing a 
Norwegian national qualification framework. A national implementation of learning 
outcomes has not yet been accomplished, but several institutions have themselves taken 
the initiative to commence this work. 

NOKUT’s regulations impose demands upon the descriptions of programme 
objectives, described as knowledge, skills, attitude and competence, upon the 
completion of the study programme. The descriptions of objectives are not placed at a 
level within a national qualification framework. 

NOKUT has evaluated learning outcomes in initial accreditations (application for 
accreditation of new study programmes or courses) and in revisions of accreditations 
(evaluation of accredited study programmes) since 2003. In initial accreditations, the 
experts evaluate whether the description of the programme objectives is reasonable. 

In revisions of accreditations, a more comprehensive and closer examination of the 
learning outcomes and competences is carried out as a part of the experts’ evaluation. 

This example describes how NOKUT uses graduate surveys and interviews with 
employers in the evaluation of learning outcomes and competences as part of the 
revisions of accreditations at bachelor and master level. Interviews with employers 
and graduate surveys, together with the institution’s self-evaluation, a site visit, and 
other written documentation such as data on student performance and fulfilment rates 
indicate the extent to which the graduates of the particular programme have reached 
the expected learning objectives. An indicator cannot be understood by itself, but has 
to be seen in relation to others. 

In Norway, the institutions themselves are responsible for the contents and the 
structure of their study programmes. This means that programmes with similar names 
may have different content and structure related to their institutional profile and the 
regional demands and challenges. Some study programmes are regulated by National 
Framework Plans. 

The use of graduate survey and interviews for the purpose of exploring 
learning outcomes 

The survey of the graduates is done in two steps: 1) a questionnaire sent to all graduates 
of specific classes; 2) an interview with 2-5 graduates. The purpose of the questionnaire 
is to ask the graduates to what extent their experience of the programme match the 
objectives the institutions have set for the programme, and the competences they have 
achieved. All objectives are listed for the graduates. Questions asked are, among others; 
“How well do the objectives match your understanding of the study programme? How 
well do the competences achieved during your study match the demands of the labour 
market?” The aims of the interviews with candidates are to gather overall descriptions. 
The graduates are for example asked to tell how they use the competence they have 
achieved at the commencement of their professional life. The graduates decide 
themselves whether they want to take part in the survey. In the questionnaire, the 


graduates indicate whether they would like to take part in an interview. Graduates for 
the interviews are selected randomly. 

Interviews are conducted with employers who have hired graduates who have 
recently completed their studies. The purpose of these interviews is to investigate 
whether the graduates upon completion of the programme have a satisfactory level of 
competence at the commencement of their first job. Questions asked are, among others: 
“Are the graduates able to start working directly after completion, or do they need 
further training? Can the graduates work independently or do they need supervision?” 
How well the graduates can participate in professional life is related to whether the 
programme honours its objectives and fulfils the competence demands from the labour 
market. 

The use of graduate surveys and interviews with the employers 

As a part of NOKUT’s internal quality assurance procedure, all experts and institutions 
are asked to complete a questionnaire about the accreditation method and the 
evaluation-process. NOKUT uses this type of evaluation as input for the further 
development of the agency’s methods and procedures. 

The evaluation of NOKUT’s use of graduate questionnaires and the interviews with 
the employers shows that the experts and the institutions find this method suitable 
and an important indicator of quality in higher educationn. Through the experts’ 
use of these methods, NOKUT also finds them suitable in evaluations. In future 
revisions of accreditations, NOKUT will assess the use of graduate surveys. Changes 
and adaptations to the method will be carried out. NIFU-STEP (NIFU-STEP report 
40/2007) has asked: “What is a good combination of indicators?” This is something 
NOKUT must decide upon on in future revisions of accreditations. 

So far, NOKUT has not decided how employers can best express their experience 
of the graduates. This kind of information is considered as essential to a general 
evaluation of the competence that students have acquired through completion of the 
study programme. 


11 Expert responses: 

81% of the experts respond that the graduate survey was used in the evaluation. 

85.7 % use graduate-satisfaction as a good indicator of quality in higher education. 

90.5 % respond that NOKUT should always undertake graduate survey in connection with revision of accreditation. 
Institution responses: 

83.3 % respond that graduate-satisfaction is a good indicator of quality in higher education. 

77.8 % respond that graduate survey is a suitable method to get information of candidate satisfaction. 

88.3 % respond that NOKUT should always undertake graduate survey in connection with revision of accreditation. 


PART D. Some considerations on the 
use of learning outcomes in the future 

Evaluating learning outcomes will become an important part of both the institutions’ 
internal quality assurance and the external quality assurance the agencies are 
responsible for. There are different kinds of balance between external and internal 
quality assurance in the five countries, and there is also variety in the methods used. 

It is not yet clear how the institutions themselves are going to evaluate whether 
the students are fulfilling the learning outcomes and how they will ensure that the 
learning outcomes are in accordance with the descriptions of the national qualification 
framework in the programmes. There are several basic questions for the agencies and 
the institutions to consider when evaluating learning outcomes in study programmes. 
These are, for example, connected to assessment criteria, how to measure learning 
outcomes, and different interpretations of the contexts in the different countries. A 
challenge is to develop indicators that measure the achievement of learning outcomes 
compared with the intended learning outcomes, and hard-to-measure variables like 
efficiency, etc., which provide limited information about the learning outcomes. 

Currently, higher education institutions, quality assurance agencies and research 
institutes worldwide are working on the topics of describing and formulating, 
measuring and evaluating learning outcomes. In the future, there will be a range of 
examples of assessment criteria and measurement indicators. Both audit and evaluation 
of study can promote enhancement of the work on learning outcomes. 


Learning outcomes in quality audits 

As higher education institutions create and introduce learning outcome descriptions 
tfor their study programmes, the next step would seem to be the inclusion of these 
outcomes in the assessment of students. The question arises as to how the extent to 
which a student acquires learning outcomes, knowledge, skills and competences can be 
assessed? Further, how can institutional performance, in terms of learning outcomes, 
be evaluated? These questions are especially relevant to quality audits. Currently, 
NOKUT and FINHEEC undertake quality audits. EVA and HSV also carry audits, but 
not on a regular basis, and at the moment. Quality audits focus on the institution’s 
internal quality assurance system. In other words, the procedures are in place to 
continuously improve teaching, research and other activities. Audits do not evaluate 
the quality of teaching or research itself, but the quality procedures and the way in 
which they are operated. Describing and evaluating learning outcomes will most likely 
become a crucial part of the institutions’ systems of quality assurance in the future. 

But to do this, the institutions must themselves develop methods to assess their own 
learning outcomes. 

Generic competences such as communication skills, teamwork, international and 
developmental skills, undertake life-long learning and attitudes are fairly diffuse 
phenomena are open to various interpretations. Therefore, the assessment of learning 
outcomes can be problematic. Higher education institutions are facing the challenge of 
how to best assess these student learning outcomes. 

Also, by assessing student acquisition of learning outcomes, the institution will 
be able to develop its teaching methods and other procedures. By gathering reliable 
information on the quality of graduating students, the institution will be better 
equipped to communicate the quality of its education to external stakeholders, again 
contributing to the success of the institution. 

Learning outcomes are best included in quality audits when the institutions 
themselves are actively assessing student learning outcomes. When this is the case, the 
inclusion of learning outcomes in quality audits would then be fairly straightforward. 

In both NOKUT’s and FINHEEC’s quality audit models, the institutions are allowed to 
create virtually any kind of quality assurance system that they see fit. However, as the 
use of learning outcomes becomes more widespread, the introduction of a new criterion 
will perhaps be necessary, specifically for learning outcomes, assuming these becomes 
a new significant field for the quality assurance of the institutions. 

The external experts who conduct the quality audits mostly come from other higher 
education institutions. As the use of learning outcomes becomes more widespread, the 
experts with knowledge and experience of working with the outcomes will be needed. 
Knowledge and experience of working with learning outcomes could therefore become 
an important theme in the training of experts. 

It seems that learning outcomes can be integrated into quality audits fairly easily. 
However, it does not seem possible or plausible for audits to evaluate whether a learning 
outcome complies with a national qualification framework, or whether a course 
or a study programme is properly designed to fulfil the learning outcomes. But the 
audit process can investigate whether the institutions themselves have good tools for 
evaluating their learning outcomes, and how they work to improve them. Audit would 
therefore promote enhancement of the work on learning outcomes. 


Learning outcomes in evaluation of 
study programmes 

EVA, HSV and NOKUT are the Nordic quality assurance agencies that currently 
evaluate courses and study programmes. All three agencies find it natural to look 
at learning outcomes in future evaluations of study programmes. EVA has already 
started this work. NOKUT undertake evaluations quite similar to evaluating learning 
outcomes. HSV and NOKUT are preparing to commence the evaluation of learning 
outcomes for courses and study programmes. 

There are many questions related to the work of developing, implementing, 
measuring and assessing methods of evaluating learning outcomes for a course or study 
programme. Discussing learning outcomes at an aggregate level is most interesting 
from the point of the quality assurance agencies. In this part, we have chosen to raise 
three questions related to a common understanding of learning outcomes, assessment 
criteria and how to measure learning outcomes. 

A common understanding of learning outcomes 

The Nordic countries are often regarded as resembling each other and being quite 
homogeneous. But the legislations differ, and there are political and cultural differences 
regarding higher education. The descriptions of status of the learning outcomes in part 
B shows different choices within higher education in the Nordic counties. Questions 
connected to a common understanding of learning outcomes are still not resolved in 
the Bologna process. So far there is a common definition of learning outcomes, but 
no terms exist of most of the other concepts related to learning outcomes. How can a 
common Nordic understanding of learning outcomes best be encouraged when they are 
open to different interpretations as they are translated into different contexts and usage 
in the Nordic countries? This question is significant for the mutual understanding and 
confidence in quality and transparency. 

At the Bologna seminar on learning outcomes in Edinburgh (2008), one of 
the concluding remarks was that there is a perceived lack of clarity and shared 
understanding about some of the key terms associated with the introduction of learning 
outcomes in different countries (e.g. “competences”, “workload” and “notional learning 
effort”), which was likely to impede effective implementation. A request was made 
for consideration to develop an agreed terminology based on a shared understanding 
among staff, students and other stakeholders about what the key concepts mean. 
Furthermore, there is uncertainty about whether learning outcomes should be written 
at “threshold”, “average” or “modal” level. It recommends that outcomes should 
normally be written at “threshold” level to facilitate recognition and mobility. Learning 
outcomes in all the Nordic countries are written at “threshold” level. 

It might be easier for the Nordic countries to reach a common understanding of the 
key terms related to learning outcomes. The work with learning outcomes has led to 
the identification of a number of terms which may need a further clarification to make 
the use of learning outcomes more comprehensible. But since the Bologna process 
is a European process, the best step towards the aim of transparency and quality 
enhancement in Europe is common European terms and definitions. 


Learning outcomes and assessment criteria 

In the Bologna process, learning outcomes were established as a tool to describe 
the outcomes at the end of a period of learning. Moreover, today, learning outcomes 
are not just an isolated tool at the level of curriculum design, but also represent an 
approach that plays a significant role in a much wider context that includes: the 
integration of academic and vocational education and training; the assessment of prior 
experiential learning; the development of lifelong learning qualification frameworks, 
the development of credit transfer and accumulation systems (Bologna Seminar on 
learning outcomes, 2004). 

Should learning outcomes be used to establish detailed assessment criteria at the 
level of the module and qualification? At the international level, learning outcomes 
represent away to communicate external reference points, a slightly different role 
than at the local and national levels. They will by definition be much broader and less 
precise than any national descriptors. For example, the European Higher Education 
Area has adopted the broad generic ‘Dublin’ descriptors as the cycle descriptors for its 
Bologna overarching qualification framework. These cycle descriptors provide a context 
to help national authorities develop their own more detailed level descriptors. Provided 
that common approaches are used by the different states within their own national 
systems, learning outcomes open up the possibility of real transparency, mobility 
and fair recognition on a scale impossible in the past. At the international level they 
aid transparency, recognition and comparability by providing common overarching 
reference points (Adams, 2008). 

Higher education in the Nordic countries is based on traditions, culture and political 
decisions, and this will probably continue to be the case in the future. Too structured 
or strict descriptions of study programmes may lead to a move towards the national or 
European standardisation of content. As Stephen Adams pointed out at the Bologna 
conference in Edinburgh (2008), learning outcomes should be seen as guides, not as 
straitjackets. At the level of study programme, subject or module, learning outcomes 
must be written in the context of appropriate national and international external 
reference points (Bologna Seminar on learning outcomes, 2004). These external 
reference points might be used for the evaluation of study programmes, subjects or 
modules. External reference points are the only opportunity for quality assurance 
agencies to assess and evaluate learning outcomes, and form a basis for transparency. 

How to measure learning outcomes? 

When evaluation is to form a basis for accreditation, the result will always indicate 
whether the object of evaluation is at a certain level or not. Quality assurance agencies 
need tools and indicators to measure learning outcomes upon completion of a study 
programme. In the section “Agency experiences of evaluating learning outcomes in 
study programmes” EVA and NOKUT share their experiences. 

There are several indicators used when assessing the graduates’ results at completion 
of a study programme. NIFU-STEP (NIFU-STEP report 40/2007) mentions for example 
grades, failure rates and retentions as indicators. In this report, NIFU-STEP states that 
these rates tell us about how productive the students and the institutions are, and little 
about the graduates learning outcomes. The combination of different indicators and the 
graduates’ self-evaluation are currently seen as the best methods of assessing learning 
outcomes. This NOQA report tells that Denmark and Norway have made different 


choices related to indicators used as basis for assessment. There are questions of validity 
and reliability related to all indicators identified so far. As the report by NIFU-STEP 
states, more precise assessment criteria and methods for measuring learning outcomes 
would probably give place focus on the results of learning. 

Above, three questions are raised, but no answers are given. The question of the 
common understanding has to be solved in a greater European context. The questions 
related to assessing and measuring learning outcomes in a national context are now 
being considered by the Nordic quality assurance agencies. 


Final reflections 


The Nordic higher education institutions are in the progress of describing learning 
outcomes and developing methods of measuring and assessing. The quality assurance 
agencies are at the same time working on developing methods for measuring, assessing 
and evaluating learning outcomes. The experience of the agencies is limited. How 
to measure learning outcomes is seen by the agencies as one of the main challenges 
for the future. The measurement question also relates to how the learning outcomes 
are used by institutions. At the present time, it seems that the principal function 
of learning outcomes at institutions is informative: to present the substance and 
properties of a study programme or an individual course in terms of what the student 
is expected to be able to do after completion. The responses to the common NOQA 
questionnaire survey of Nordic higher education institutions reflected the importance 
of being able to illustrate contents of education in this way. Equally interesting are the 
other functions which many institutions have imposed on learning outcomes. The 
outcomes help to adjust examination criteria and in some institutions, the learning 
outcomes are used in measuring institutional performance and as a part of the quality 
assurance systems. Since grading has previously dwelt largely on acquisition and 
ability to use knowledge on a given subject, the introduction of learning outcomes can 
potentially mark a fundamental shift in the assessment of students. Learning outcomes 
enable institutions to assess students also in terms of generic competences and skills, 
such as communication, teamwork, critical thinking and so on. There is little doubt 
about the relevance of these abilities to working life and the importance of being able to 
measure them. 

To higher education institutions, learning outcomes can present an opportunity 
to present what their students are actually capable of, and what they can do after 
graduation; not merely what they know or are supposed to know. By creating tools and 
mechanisms for measuring student learning outcomes, institutions can find new ways 
to develop pedagogical methods and teaching. 

To students, learning outcomes present an opportunity to be informed on study 
programme content to a greater degree than before. If well described, the outcomes 
tell a prospective student far more than the previous course descriptions, which have 
traditionally dealt primarily with the knowledge to be acquired. 

To ministries of education, learning outcomes also present interesting possibilities. 
The present governments in many Nordic countries are looking into ways in which 
quality of education could be included more fundamentally in the performance criteria 
of higher education institutions. With national qualification frameworks and learning 
outcomes described in every institution, ministries can set performance targets in 
terms of learning outcomes and direct funding according to these. This situation is 
still far away, but it seems potentially plausible to use learning outcomes in this way. 
However, the Nordic countries must first continue to develop objective and credible 
ways in which to evaluate learning outcomes. 

National qualification frameworks based on learning outcomes are being 
implemented in all Nordic countries with a view to completion by 2010. It might be 
an idea for a future Nordic joint project to look at how to measure learning outcome, 
once this has been implemented in all countries and we have more experience with 


it. We have seen from previous NOQA projects that the sharing of experiences and 
the exchange of viewpoints has provided useful insight to quality assurance. A future 
NOQA project on measuring learning outcomes could provide useful and practical 
contributions to the further development of evaluation methods, both for the 
institutions and for the quality assurance agencies. 
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Appendices 

Appendix 1: 

QUESTIONS TO THE AGENCIES 

1. What, if any, is your agencies official view on the use of learning outcomes in 
evaluations? 

2. To what extend is learning outcomes connected to your national qualification 
framework /European qualification framework? 

3. Have your agency had any part in introducing and developing the concept of 
national qualification framework? 

4. What is the current situation regarding the implementation of learning outcomes 
in your country? (Give a brief summary of the information you already have sent 
to the group) 

5. What kind of experience does your agency have of using learning outcomes or 
equivalent in your evaluations? 

6. What kind of evaluations does your agency undertake to day, and which of them 
do you think should be revised to include learning outcomes in the future (before 
2010)? 


Appendix 2: 

QUESTIONS TO THE INSTITUTIONS 

1. To what extent have you introduced learning outcomes (or description of 
competences, competence goals, knowledge, skills etc.) in the study programmes 
of your institution? 

2. How were the learning outcomes formulated? Which groups (teachers, students 
other stakeholders) were involved in the process of creating the learning outcomes 
in your institution? 

3. How does your institution define the concept of learning outcomes? What are the 
learning outcomes for you? 

4. Did you use the national qualifications framework or the European qualifications 
framework in the process of formulating learning outcomes? 

5. Where did the initiative to describe learning outcomes come from? 

6. How do you use the descriptions of learning outcomes in your institution? 

7. Assess how useful the descriptions of learning outcomes are for your 
institution? 

8. How have you introduced the learning outcomes to students, teachers and other 
stakeholders? 

9. Do you have a plan of following up or improving the descriptions of learning 
outcomes? If yes, please describe it. 
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Appendix 3: 

THE EDUCATION SYSTEMS IN THE NORDIC COUNTIES 

Figures 1-5 in part B, are based on the following original figures. 

1) Denmark: 


HIGHER EDUCATION IN DENMARK 



PhD programmes 
(normally 3 years) 


+ + + 



Bachelor's (BA/BSc) 
programmes 


UNIVERSITY SECTOR 





Academy Profession 
programmes 


* 

j 

■f COLLEGE SECTOR 


Master's 

programmes 

Diploma 

programmes 

VVU 

programmes 

Adult 

education 


General and vocationally oriented upper secondary education 
(12-13 years of education) 


Vocational education and training 
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2) Finland: 


ISCED- 

classification 


THE FINNISH EDUCATION SYSTEM 



4-5 


1-3 


1-3 


1 


1-9 

A 


Duration 
in years 


DOCTORAL DEGREES 
- licenciate degrees 


MASTER'S DEGREES 


POLYTECHNIC 
MASTER'S DEGREES 


Work experience 3 years ^ 


BACHELOR'S DEGREES 
Universities 


POLYTECHNIC 
BACHELOR S DEGREES 
Polytechnics 


MATRICULATION 
EXAMINATION 
general upper secondary 
schools 


VOCATIONAL 
QUALIFICATIONS 
vocational institutions and 
apprenticeship training 





Work 

experience 

r ^ 

Additional basic education 

1 

BASIC EDUCATION 
(comprehensive schools) 7-16 

-year-olds 

Pre-primary education, 6-year-c 

Ids 

U 


ISCED-CLASSIFICATION 

0 Pre-primary education 

1 Primary education or first stage of basic education 

2 Lower secondary or second stage of basic education 

3 (Upper) secondary education 

4 Past secondary nontertiary education 

5 First cycle of tertiary education 

6 Second cycle of tertiary education 









COMPULSORY EDUCATION 
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3) Iceland: 


STRUCTURE OF THE EDUCATIONAL SYSTEM 
ICELAND 


Age 

28 

27 

26 

25 

24 

23 

22 

21 

20 

19 

18 

17 

16 

15 

14 

13 

12 

11 

10 



COMPULSORY SCHOOLS 
(Grunnskolar) 


PRE-PRIMARY SCHOOLS 
(Leikskolar) 


Open access 

Access to specific programmes 


There are four levels of education in Iceland: pre-primary schools, 
compulsory (single-structure - primary and lower secondary 
education), upper and higher education 


4) Norway: 


THE STRUCTURE OF THE NORWEGIAN EDUCATIONAL SYSTEM AND DEGREES 


LOWER SECONDARY SCHOOL 
(compulsory) grade 8-10 





PROFESSIONALLY 
ORIENTED DEGREES/ 
QUALIFICATIONS 
6 years 

ONE-TIER MASTER S 
DEGREES 
5 years 

MASTER'S DEGREES 

2 years 

BACHELOR'S DEGREES 

3 years 


UPPER SECONDARY SCHOOL 
vocationally oriented academically oriented 


LOWER SECONDARY SCHOOL 
(compulsory) grade 8-10 


PRIMARY SCHOOL 
grade 1-7 (from 1997: age 6-12) 


6 

5 

4 

3 

2 

1 


14 

13 

12 

11 


10 

9 

8 

7 

6 

5 

4 

3 

2 

1 


Exceptions not included in the diagram: 

Master's degree in architecture from Oslo School of Architecture and Design, Oslo: 5 years 
Master's degrees of 1-1 years duration 
Bachelor's degrees of 4 years duration (music) 

General Teacher Training: 4 years 
Hogskolekandidat degree: 2 years 
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5) Sweden: 


Year 

4 

3 

2 

1 




_3 

2 

1 


Third level 


Degree of 
Doctor 


Degree of 
Licenciate 


Degree of 
Master 
(one year) 


Degree of 
Master of 
Arts 

(one year) 


Second level 


I 

Degree of Degree of 

Master Master of Arts 

(two years) (two years) 

x ■ 


First level 


University- 

diploma 


University- 
diploma in ' 
Arts 


Degree of 
Bachelor 

Degree of 
Bachelor of 
Arts 

i 

k 



Professional 

degrees 


Secondary education 


Exceptions not included in the diagram: 

Master's degree in architecture from Oslo School of Architecture and Design, Oslo: 5 years 
Master's degrees of 1-1 years duration 
Bachelor's degrees of 4 years duration (music) 

General Teacher Training: 4 years 
Hogskolekandidat degree: 2 years 


Higher education 






The present report is the outcome of the project conducted by the Nordic Quality 
Assurance Network in Higher Education (NOQA) in 2007-2008 entitled “Learning 
outcomes: Common framework - different approaches to evaluating learning 
outcomes in the Nordic countries”. ENQA publishes the report as part of a series of 
occasional papers that deal with developments in European quality assurance. 



